Description

This PR updates our chunking logic for Dataset create_data_rows and upsert_data_rows to chunk by size of file, not the number of data rows. This will result in generating files of similar size, making processing more predictable and reducing a chance of error or a performance issue due to a very large file

We have set the file size limit to 10MB based on the following considerations:

• in an average case, each row in the files consists of a hash with row_data and, possibly, global or external key. In our estimate the json-encoded size of such a hash is around 500 bytes
• current chunk limit of a file is 10K lines
• we have set the chunk size for a file to be 10MB

Fixes # (issue)

Type of change

Please delete options that are not relevant.

• Internal optimization

All Submissions

• Have you followed the guidelines in our Contributing document?
• Have you provided a description?
• Are your changes properly formatted?

New Feature Submissions

• Does your submission pass tests?
• Have you added thorough tests for your new feature?
• Have you commented your code, particularly in hard-to-understand areas?
• Have you added a Docstring?

Changes to Core Features

• Have you written new tests for your core changes, as applicable?
• Have you successfully run tests with your changes locally?
• Have you updated any code comments, as applicable?